PyraPVConv: Efficient 3D Point Cloud Perception with Pyramid Voxel Convolution and Sharable Attention

Designing efficient deep learning models for 3D point cloud perception is becoming a major research direction. Point-voxel convolution (PVConv) Liu et al. (2019) is a pioneering research work in this topic. However, since with quite a few layers of simple 3D convolutions and linear point-voxel feature fusion operations, it still has considerable room for improvement in performance. In this paper, we propose a novel pyramid point-voxel convolution (PyraPVConv) block with two key structural modifications to address the above issues. First, PyraPVConv uses a voxel pyramid module to fully extract voxel features in the manner of feature pyramid, such that sufficient voxel features can be obtained efficiently. Second, a sharable attention module is utilized to capture compatible features between multi-scale voxels in pyramid and point cloud for aggregation, as well as to reduce the complexity via structure sharing. Extensive results on three point cloud perception tasks, i.e., indoor scene segmentation, object part segmentation and 3D object detection, validate that the networks constructed by stacking PyraPVConv blocks are efficient in terms of both GPU memory consumption and computational complexity, and are superior to the state-of-the-art methods.


Introduction
With the advance of depth sensing devices, 3D point clouds can be captured in a much easier manner. erefore, applications of 3D point cloud perception are now booming, e.g., simultaneous localization and mapping (SLAM) [1][2][3], and autonomous driving [4][5][6]. In the last few decades, 3D point cloud perception mainly depends on hand-crafted shape descriptors [7][8][9]. Until very recently, researchers start to extend deep learning models which are mature in the field of 2D computer vision to handle 3D perception tasks, significantly refreshing the state-of-the-art records [10][11][12]. However, 3D deep learning models with higher accuracies always have higher complexities, which now becomes the major obstacle for their application to real-world scenarios.
ere have been many attempts that utilize deep networks to perceive 3D point clouds. According to the representation of data that feeds to deep networks, these methods could be divided into two categories broadly: structure-based methods and point-based methods. Structure-based methods first convert irregular point clouds into structured grid representations, e.g., projecting point clouds into bird's eye views [13,14] or rasterizing into 3D voxel grids [10,[15][16][17], and then adopt traditional 2D convolutional neural networks (CNNs) or their simple extensions to extract discriminative CNN features. However, it introduces exponential computational cost and memory for detailed 3D geometric learning at high resolutions. Point-based methods instead impose multi-layer perceptrons (MLPs) followed with maximum pooling [11,18] or irregular kernels [19] to 3D point clouds, such that they can handle points directly. Nevertheless, since these methods require accessing irregularly scattered points especially for local feature aggregation, they are also inefficient.
By analysing both the advantages and disadvantages of structure-based and point-based methods, the pioneering point-voxel convolution (PVConv) [20] proposes to combine them together for efficiency purpose. Particularly, point-based MLPs are adopted to extract 3D features in the point branch; and voxel convolutions aggregate local features coarsely in the voxel branch; in addition, linear interpolation is conducted to fuse the features of the two branches. PVConv brings large improvements in terms of GPU memory consumption and computational efficiency, but the accuracy of PVConv is somewhat sacrificed. By going through the structure of PVConv, we hypothesize that there are two key factors that limit its performance. First, the voxel-based networks with strong feature extraction capabilities are utilized too conservatively, i.e., with only two layers of simple 3D convolutions. Second, the voxel features that are already insufficient would further lose during the fusion process, since it is implemented via a linear-based interpolation that is not powerful enough.
To resolve the above two issues, we intentionally design a novel pyramid point-voxel convolution (PyraPVConv) block. First, we adopt a more powerful 3D voxel convolution branch that extracts multi-scale voxel features in the form of feature pyramid, such that powerful 3D voxel convolutions can be fully exploited with sacrificing moderate additional computational overhead. Second, to alleviate the information loss during the feature fusion process between point and voxel branches, we propose utilizing the attention mechanism to learn to combine them in a more compatible way. Particularly, we design a sharable attention module that learns the relevant scores between multiple voxel branches and the point branch with sharing a structure, such that it reduces the overhead required by multibranch attention. With these two designs against the weaknesses of PVConv, the PyraPVConv block can perceive 3D point clouds in a better tradeoff between accuracy and efficiency.
Overall, our contribution is three-fold.
(i) We propose a lightweight 3D perception block, PyraPVConv, that can perform 3D point cloud perception accurately and efficiently (ii) We design a voxel pyramid module, which better extracts voxel features without introducing too much additional computational overhead (iii) We devise a sharable attention module that fuses the features of point branch and multiple voxel branches in a more effective nonlinear manner We construct PyraPVCNN by stacking multiple Pyr-aPVConv blocks following PVCNN [20], and evaluate it on various point cloud perception tasks, e.g., indoor scene and object part segmentation, 3D object detection. Extensive experiments demonstrate the superiority of PyraPVConv to the state-of-the-art methods in terms of both accuracy and efficiency.

Efficient Deep Learning for Point Clouds.
Deep learning techniques have been widely adopted for handling 3D point cloud perception tasks, e.g., classification and segmentation, by projecting sparse point clouds into compact semantic representations [21][22][23][24][25]. However, as indicated in [20], most current 3D deep learning methods for point clouds are less efficient, e.g., voxel-based methods require large amounts of memory for maintaining detailed 3D structures, and pointbased methods require high CPU latency to search neighborhood points for feature aggregation.
To perceive 3D point clouds in a more efficient way, Liu et al. [20] proposed the point-voxel convolution (PVConv), that combines point-based MLPs for individual point feature extraction and voxel-based CNNs for neighborhood feature aggregation. Since with only two layers of simple 3D CNNs, the performance of PVConv is somewhat sacrificed. To better utilize the voxel information, Tang et al. [26] further designed sparse point-voxel convolution (SPVConv), which uses sparse convolution to handle high-resolution voxels on the basis of PVConv, and further adopted network architecture search (NAS) for searching the best architecture. Although their method showed good performance in largescale scenarios, sparse convolution is complex and is less efficient in common scenes. Our method also aims to fully utilize 3D voxel information. Differently, we adopt the concept of feature pyramid to balance the accuracy and efficiency.
We also notice some other works that also aim to efficient point cloud learning by randomly key point sampling [27], by irregularly volume partition [28], and by leveraging mature 2D methods [29], etc., but these directions are out of our scope.

Multiscale Feature Modeling.
Modeling multi-scale features has been validated to be a useful strategy in computer vision, e.g., maintaining scale-invariant property in SIFT [30] and controlling the receptive field in CNNs [31].
is strategy is also widely adopted in 3D deep learning. PointNet++ [18] extracts multi-scale features of point clouds by hierarchically applying PointNets [11]. 3D object detection frameworks [17,32,33] adopt different detection heads with multi-scale feature maps to handle both large and small object classes. ContFuse [34] uses continuous convolution to aggregate multi-scale feature maps from different ResNet blocks [35]. By extending RPN-FPN module [36] to 3D, Voxel-FPN [37] uses feature pyramid to aggregate voxel features of different voxelization resolutions. We also utilize the multi-scale feature modeling strategy, but are to balance the accuracy and efficiency of the voxel branch for pointvoxel convolution, which has not been investigated before.

Attention Mechanism.
Attention is originally a physiological mechanism that describes the phenomenon that humans' perception system could focus on the object of interest while suppressing the background [38]. Inspired by it, deep learning researchers attempt to exploit it to analyze networks' focus [39,40] or enforce neural networks to focus on more important features [41,42].
Attention is also widely adopted to model the relationship between two instances, e.g., generating the most relevant sentences for images in the task of image caption [43] or searching the most relevant sentences between two different languages in the task of machine translation [44]. By projecting the query instance into the same high-dimensional space as the target, relevant spatial regions of the query will be highlighted to guide the desired inference. We also aim to model the relationship. Differently, our inferred relevance is used for the fusion of features extracted from the voxel branch and point branch during point-voxel convolution.

Method
In this section, we will first review the architecture of PVConv, analyze the factors that may limit its performance and then introduce the overview of our solution, i.e., pyramid point-voxel convolution (PyraPVConv). After that, we describe the two main components of PyraPVConv: the voxel pyramid module and the sharable attention module. [20]. Methodology of PVConv Given an unordered point set P � (p k , f k ) with p k denoting the point coordinates and f k as the point features, PVConv adopts an isolated point branch to extract individual point features using MLPs similar to PointNet. Apart from that, another voxel branch in PVConv is utilized to facilitate efficient and powerful neighbourhood feature aggregation. Particularly, the feeded voxels to the voxel branch are generated by first normalizing the point coordinates and then conducting voxelization to transform the normalized point cloud (p k , f k ) into a volume V by averaging all features f k whose normalized coordinates p k fall into that voxel grid.

Review of PVConv
Finally, the two-branch features are linearly fused, i.e., the voxel features are devoxelized to the point cloud domain using trilinear interpolation, and then added with point features.
Discussion on PVConv As mentioned in PVConv, the MLPs in the point branch can already output discriminative features for each point. erefore, the main contribution of PVConv is to leverage the voxel branch for neighborhood feature aggregation.
However, perhaps focusing too much on the efficiency factor, PVConv is conservative in utilizing the voxel branch, e.g., very limited volume resolutions and 3D convolution layers are adopted. Indeed, 3D convolutions that are derived from mature 2D convolution techniques are powerful, and have a good tradeoff between accuracy and efficiency. Furthermore, simple trilinear interpolation would lead to information loss of those voxel features that are not sufficient originally.

Overview of PyraPVConv.
To resolve the above issues of PVConv, we propose a novel PyraPVConv block with two key structural modifications: a voxel pyramid module for the voxel branch and a sharable attention module for feature fusion. Similar as in PVConv, the point branch of Pyr-aPVConv extracts individual point features. At the same time, the voxel branch of PyraPVConv adopts the voxel pyramid module to extract more powerful voxel features in pyramid. en, the shareable attention module fuses point features and voxel features in pyramid nonlinearly. Please refer to Figure 1 for a demonstration.

Voxel Pyramid Module.
To extract sufficient voxel features without bringing too much additional computational overhead, we propose a voxel pyramid module to capture multiscale features in different pyramid levels, inspired by [36]. Please refer to Figure 2 for a demonstration.
Volume Generation Given the normalized point cloud (p k , f k ) as in PVConv, we generate a volume by averaging all features f k whose coordinates p k fall into that voxel grid via the following equation: where r denotes the volume resolution, N r,u,v,w denotes the number of points falling in the voxel grid (u, v, w) of volume V r , f k,c denotes the c th channel corresponding to p k , and I is a binary indicator function. Bottom-up Feature Extraction Given V r , we feed it to the same 3D voxel convolution networks as in PVConv, i.e., two groups of conv3d, batch normalization and activation layers, Unlike PVConv that adopts a conservative strategy in the voxel branch, we suggest making full use of the features in volume V r . Considering the efficiency, we conduct an average pooling operation with a scaling rate of 1/2 to f V r , and then feed it another 3D voxel convolution networks, We apply the same operations iteratively to obtain multiscale 3D voxel features. In this paper, we adopt three-scale features f V r , f V r/2 , and f V r/4 . To-down Feature Aggregation Given each 3D voxel feature, we enhance it by aggregating with the feature of a higher pyramid level (if has) that is spatially coarser but semantically strong. It is implemented by applying an upsampling operation followed with addition, (4)  4 and point features f k as inputs, we fuse them via using the attention mechanism. Since voxel features in different pyramid levels contain discriminative but complementary information, we apply attentive fusion of them with point features separately. It is implemented by converting the voxel features into the same feature space as point features, and then summarize them. See Figure 3 for a demonstration. In the following, we will describe the attention method taking f V r as an example. Attentive Fusion for a Pyramid Level Instead of fusing all voxel features, we only consider eight neighboring voxel features f V r (k, m) with m � 1, 2, . . . , 8 for each point (p k , f k ) following PVConv, considering the tradeoff between efficiency and performance. Specifically, we adopt a similar attention process to that in [45] with three key steps. First, we project point feature f k and its eight neighboring voxel features f V r (k, m) into the same feature space using two networks Q r and K r . Second, we calculate the relevant scores between eight voxel features and that point feature by conducting dot product of their flattened features. Note that, we also adopt the Sigmoid operation to enforce the relevant scores to be between 0 and 1.

Sharable Attention Module. With enhanced voxel features
Rel Finally, we convert the voxel features f V r (k, m) into a summable space with point features using the network H r , and then aggregate them weighted with the relevant scores Rel(f k , f V r (k, m)).
Multiple Pyramid Fusion with Sharable Attention For voxel features in all three pyramid levels, we apply the above attentive fusion method, and then summarize them together.
erefore, f k ′ is the final fused feature that contains both point and voxel features. Particularly, since the inputs for networks Q t with t � r, r/2, r/4 are exactly the same, we simply use one network Q and share it with all three pyramid levels for efficiency purpose.

Experimental Results
To validate the effectiveness and efficiency of PyraPVConv, we extensively evaluate the performance of PyraPVCNN, which is constructed by stacking multiple PyraPVConv blocks, on three different tasks, i.e., indoor scene segmentation, object part segmentation and 3D object detection.

Implementations.
For PyraPVCNN, we replace all the PVConv blocks in PVCNN with PyraPVConv and use the same decoding layer. We implement PyraPVCNN and reproduce all the evaluated networks with PyTorch for fair comparisons, and report the latency and memory

Indoor Scene Segmentation. Setups
We conduct semantic segmentation experiments on the large-scale Stanford 3D semantic parsing dataset (S3DIS) [47,48]. S3DIS is scanned across 271 rooms in 6 areas with a Matterport camera (combined with 3 structured light sensors at different intervals), and then each point in the scanned point cloud is annotated with a semantic label (e.g., a total of 13 objects such as chairs, tables, floors, walls, etc.). We adopt the same data processing procedure as PVCNN [20]. We train the models on regions 1, 2, 3, 4, and 6 and test them on region 5 to evaluate the mean intersection-over-union (mIoU) and the mean class-wise accuracy (mAcc), with both metrics measured as percentages. Models Six state-of-the-art methods are adopted as baselines: PointNet [20], PointNet++ [18], DGCNN [25], PointCNN [46], PVCNN [20] and RandLA-Net [27]. Since our focus is on the efficiency of deep networks, similar as in PVCNN, we also evaluate the narrower versions of PVCNN and PyraPVCNN by reducing the number of feature channels from that of the original version (denoted as 1 × C) to 12.5% (denoted as 0.125 × C), 25% (denoted as 0.25 × C) and 50% (denoted as 0.5 × C).Comparison Results e results reported in Table 1 show that Ours (0.125 × C) performs better than two state-of-the-art point-based methods: PointNet [11] and DGCNN [25] and voxel-based methods: PVCNN (0.25 × C) [20], and is comparable to the state-of-the-art RandLA-Net [27], but with less latency and GPU memory consumption. In addition, the performance of Ours (1 × C) is nearly 10% higher than that of DGCNN [25], but the latency is reduced by 20%, and GPU memory consumption is reduced by 1.6 times. In particular, Ours (0.25 × C) outperforms the full version of PVCNN, but the latency and GPU memory consumption are much less. Please refer to Figure 4 to see an illustrative demonstration of the tradeoff between accuracy (mIoU) and the incurred overhead (the number of parameters, latency and GPU memory consumption).
We also visualize the segmentation results in Figure 5. It could be seen that Ours (0.25 × C) can better utilize neighborhood information to improve the prediction of point labels, compared with PVCNN.
Ablation Studies To validate the importance of the two key modules of PyraPVConv: voxel pyramid module and the sharable attention module, we report the results of ablation studies using two narrow versions of the PyraPVCNN (i.e., 0.25 × C and 0.5 × C) in Table 2. It could be seen that the performance will drop, if we delete any one of them. In particular, the voxel pyramid module has a slightly larger impact on the performance.

Object Part Segmentation. Setups
We choose the ShapeNet Parts dataset [49] to conduct the experiment of object part segmentation. ShapeNet Parts is a collection of 16681 point clouds selected from 16 categories of the ShapeNetCore, and is manually annotated with a total of 50 parts. Following [46], we train the models on 14006 point clouds, evaluate the part-averaged IoU for each of the remaining 2874 point clouds, and then average them as the final metric, i.e., mIoU (%).
Comparison Results As shown in Table 3, Ours (0.25 × C) performs much better while the latency is 36 times lower and the GPU memory consumption is 11.7% lower than that of 3D-UNet. Moreover, it is superior to the most advanced pointbased methods such as PointNet [11], PointNet++ [18] and DGCNN [25] in all aspects. In particular, we would like to emphasize that Ours (0.5 × C) outperforms PVCNN [20], and Computational Intelligence and Neuroscience   Computational Intelligence and Neuroscience Ours (1 × C) outperforms PointCNN [46] with only about 58% latency and GPU memory consumption. Please refer to the visualization results in Figure 6 for intuitive comparisons.

3D Object Detection. Setups
We conduct 3D object detection experiments on the large outdoor dataset KITTI [51]. KITTI provides 7481 training and verification samples for objects such as cars, pedestrians, and cyclists, of which 3711 samples are used for training and the remaining 3769 samples are for verification. We evaluate all models 20 times and report the average 3D average accuracy (AP) measured as percentages.

Conclusion
In this paper, we have presented a novel PyraPVConv block with the aim of efficient 3D point cloud perception. e key idea is to utilize voxel pyramid to make full use of voxel information and then adopt the attention mechanism for better point-voxel feature fusion. Extensive experiments validate the superiority of PyraPVConv. We hope that PyraPVConv can act as an important part of various deep networks for 3D point cloud perception. In the future, we plan to utilize the technique of neural architecture search for designing more efficient network architectures.

Data Availability
All datasets used in this paper are publicly available.

Conflicts of Interest
e authors declare that they have no conflicts of interest.

Authors' Contributions
Yuhong Chen and Weilong Peng contributed equally to this work.